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Detailed Action 



Specification 

Objections 

1. The disclosure is objected to because of the following informalities: 

a. On page 1, line 3 1 of the Applicant's disclosure, the word "rage" should be replaced with the word 
"range". 

b. The sentence on page 1, lines 31-35 of the Applicant's disclosure should be revised to correct the 
apparent grammatical errors. 

c. On page 2, line 16 of the Applicant's disclosure, the phrase "like in a good whether" should be 
replaced with "like in good weather". 

d. The sentence on page 3, lines 32-34 of the Applicant's disclosure ("The sight-axis switch ... on 
the route") should be revised to correct the apparent grammatical errors. 

e. On page 4, lines 18-19 of the Applicant's disclosure, the phrase "an intensifier that responses to 
faint light" should be changed to correct the apparent grammatical error. For example, change the 
phrase to "an intensifier that is responsive to faint light". 

f. On page 7, line 37 of the Applicant's disclosure, the word "stere-image" should be changed to 
"stereo- image". 

g. On page 8, line 2, of the Applicant's disclosure, the word "rout" should be changed to "route". 
Appropriate correction is required. 
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Claims 

Objections 

2. Claims 3-4 are objected to because of the following informalities. Claims 3 and 4 refer to vision data. In 
order to maintain consistency among the claims, the Applicant should change vision data with view data. 
Appropriate correction is required. 

Rejections Under 35 U.S.C. § 1 12(2) 

3. Claim 6 is rejected under 35 U.S.C. 1 12, second paragraph, as being indefinite for failing to particularly 
point out and distinctly claim the subject matter which applicant regards as the invention. Claim 6 proposes that one 
of the stereo cameras be a milli-wave camera. Given the wavelength in which these systems operate, MMW cameras 
do not particularly lend themselves to being stereo cameras. Taking this into account, claim 6 will be interpreted, 
henceforth, as: "The integrated vision system according to claim 1 further comprising at least a first stereo-camera, a 
second camera and a third stereo camera, the first stereo camera being an infrared camera, the second camera being 

a milli-wave camera and the third stereo camera being an intensifies the first, the second and the third stereo- 
cameras being selectively used in accordance with actual views". 

Rejections Under 35 U.S.C. § 102(e) 

4. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the 

rejections under this section made in this Office action: 

(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed in the United States 
before the invention by the applicant for patent or (2) a patent granted on an application for patent by another filed in the United States 
before the invention by the applicant for patent, except that an international application filed under the treaty defined in section 351(a) 
shall have the effects for purposes of this subsection of an application filed in the United States only if the international application 
designated the United States and was published under Article 21(2) of such treaty in the English language. 

5. Claims 1 and 4 are rejected under 35 U.S.C. 102(e) as being anticipated by Yasui et al (U.S. Patent 
6,483,429). 
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6. The following is in regard to Claim L Yasui et al. disclose a parking assistance system (obstacle detection 
sytem) mounted in a vehicle (Yasui et al. Abstact). The parking assistance system generates 3D data corresponding 
to the obstacle and displays this to the user or passenger. In this way, the parking assistance system can be 
considered an integrated vision system, in the same vein as the Applicant's integrated vision system. Yasui et ai.'s 
system includes: 

(1 .a.) At least one stereo-camera installed in a vehicle for taking images of predetermined 
outside area (e.g. camera 10 in Yasui et al. Fig. 1). 

(1 .b.) Stereo-image processing of a pair of images taken by the stereo-camera to recognize 
objects that are obstacles to the front, thus generating obstacle data.. See Yasui et al. 
column 2, lines 38-64. The components of Yasui et al.'s system executing this process 
would, therefore, constitute a stereo-image recognizer. 

(1 .c.) Generating three-dimensional view data based on the pair of images taken by the stereo- 
camera and the obstacle data from the stereo-image recognizer. See Yasui et al. column 
2, lines 38-64, column 3, lines 30-62, and Fig. 3. The components of Yasui et al.'s 
system that generate this data can be regarded as the view data generator. These 
components are integrated in the sense that they are part of the parking assistance 
system. 

(Ld.) An image display (e.g. display 30 in Yasui et al. Figs. 1-2) for displaying the view data, 
generated according to step (lx), as visible images to crew on the vehicle. See Yasui et 
al. column 4, lines 18-21. This display is integrated in the sense that it is part of the 
parking assistance system. 

It has thus been shown that the parking assistance system of Yasui et al. represents an integrated vision system that 
conforms to the system set forth in claim 1. In this way, Yasui et al. anticipates the claimed integrated vision system 
of claim 1. 

7. The following is in regard to Claim 4. As shown above the parking assistance system of Yasui et al. can be 
considered an integrated vision that conforms to all limitations of claim 1. The system of Yasui et al. further 
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includes the capability of removing the three-dimensional (3D) vision data from the integrated vision data. See 
Yasui et al. column 10, lines 15-24. That is, 3D data relating to the detected obstacle is removed from the displayed 
image (a).It has thus been shown that the parking assistance system of Yasui et al. represents an integrated vision 
system that conforms to the system set forth in claim 4. In this way, Yasui et al. anticipates the claimed integrated 
vision system of claim 4. 

8. Claims 1 and 5 are rejected under 35 U.S.C. 102(e) as being anticipated by Strumolo et al. (U.S. Patent 
6,535,242). 

9. The following is in regard to Claim 1. Strumolo et al. disclose a system for acquiring and displaying 
vehicular information - more particularly, 3D information of objects and/or the environment surrounding the 
vehicle. As with Yasui et al., this system can be considered an integrated vision system in the same vein as the that 
of the Applicant. Strumolo et al/ system includes: 

(La.) At least one stereo-camera installed in a vehicle for taking images of predetermined 

outside area (e.g. stereo camera pairs 18, 20 and 22, 24 depicted in Strumolo et al. Fig. 
1). 

(1 .b.) Stereo-image processing of a pair of images taken by the stereo-camera to recognize 
objects that are obstacles to the front, thus generating obstacle data. Since the stereo 
cameras capture stereo images of objects (e.g. obstacles) in front of the vehicle (Strumolo 
et al. column 3, lines 61-64) and displays to the user a 3D representation of the capture 
objects (Strumolo et al. column 2, lines 26-36), stereo-image/stereo-vision processing of 
stereo-image pairs is an inherent feature of the system of Strumolo et al. The components 
of Strumolo et al.'s system performing the stereo-image processing would, therefore, 
constitute a stereo- image recognizer. 

(1 ,c.) Generating three-dimensional view data based on the pair of images taken by the stereo- 
camera and the obstacle data from the stereo-image recognizer. See Strumolo et al. 



Application/Control Number: 09/866,773 Page 6 

Art Unit: 2623 

column 2, lines 9-36. Note that the display of 3D data of objects, acquired from the stereo 
images captured by the stereo cameras, necessarily entails the generation of 3D view 
data. Furthermore, stereo-image processing, such as in step (l.b) genererally involves the 
generation of 3D view data from pairs of images. The components of Strumolo et al.'s 
system that generate this data can be regarded as the view data generator. These 
components are integrated in the sense that they are part of Strumolo et al.'s system. . 
(l.d.) An image display (e.g. display 12 in Strumolo et al. Figs. 1) for displaying the view data, 
generated according to step (l.c), as visible images to crew on the vehicle. See Strumolo 
et al. column 2, lines 26-36. This display is integrated in the sense that it is part of the 
system of Strumolo et al. 

It has thus been shown that the vehicular information system of Strumolo et al. represents an integrated vision 
system that conforms to the system set forth in claim 1. In this way, Strumolo et al, anticipates the claimed 
integrated vision system of claim 1. 

10. The following is in regard to Claim 5. As shown above the parking assistance system of Strumolo et al. can 
be considered an integrated vision that conforms to all limitations of claim 1. Furthermore, the stereo-cameras of 
Strumolo et al.'s system include two infrared cameras arranged as separated from each other by a distance 
corresponding to a specific base line. See Strumolo et al. column 3, lines 57-67 to column 4, lines 1-18 and notice 
reference numbers 18, 20 and 22, 24 in Fig. 1. Therefore, the vehicular information system of Strumolo et al. 
represents an integrated vision system that conforms to the system set forth in claim 5. In this way, Strumolo et al. 
anticipates the claimed integrated vision system of claim 5. 

Rejections Under 35 U.S.C. § 103(a) 

11. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set 
forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if 
the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would 
have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 
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12. Claims 1-3 are rejected under 35 U.S.C. 103(a) as being unpatentable over Sato (U.S. Patent 6,445,815) in 
view of Hamilton et al. (U.S. Patent 5,296,854). 



13. The following is in regard to Claim 1. Sato discloses an augmented reality presentation system, with a 
specific application to collision detection (Sato column 17, lines 32-38), where 3D images are generated from 
sequences of stereo images and presented to the user via a head-mounted display (HMD). The augmented reality 
system (ARS) generates 3D data corresponding to an object in the view of a user (e.g. an obstacle) and displays this 
to the user or passenger. In this way, the ARS can be considered an integrated vision system, in the same vein as the 
that of the Applicant. Sato's ARS includes: 

( 1 .a.) At least one stereo-camera for taking images of predetermined outside area (e.g. camera 

102L and 102R depicted in Sato. Fig. 1). 
( 1 .b.) Stereo-image processing of a pair of images taken by the stereo-camera to recognize 
objects that are obstacles to the front (Sato column 17, lines 32-38), thus generating 
obstacle data. See, for example, column 10, lines 20-47 and column 17, lines 32-38. The 
components of Sato's system performing the stereo-image processing would, therefore, 
constitute a stereo-image recognizer. 
(1 .c.) Generating three-dimensional view data based on the pair of images taken by the stereo- 
camera and the obstacle data from the stereo-image recognizer. See, for example, Sato 
column 7 lines 9-16, column 12 lines 46-50, and column 16 lines 36-51. The image 
generation module 300 of the various embodiments of Sato's system can be considered 
an integrated view data generator. 
(1 .d.) An image display (e.g. see-through HMD 100 in Sato Fig. 16) for displaying the view 
data, generated according to step (l.c). 
Sato suggest that this ARS be used in a vehicle, namely a moving robot (Sato column 17, lines 39-43). Despite this, 
Sato does not show or suggest that the vehicle be a human or crew-carrying so that view data is displayed to a crew 
on the vehicle. (The HMD and the moving robot would presumably be separate in the augmented reality system of 
Sato). 
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14. Hamilton et al. discuss the application of a virtual image display system, similar to the ARS of Sato, on 
board a crew-carrying vehicle, namely a military helicopter. See Hamilton et al. Figs. 1-3 and 7 and column 4, lines 
1-60, column 14 lines 49-68 and column 15 lines 1-37. 

15. The teachings of Hamilton et al. and Sato are combinable because they are analogous art. Specifically, the 
virtual image display systems of Hamilton et al. and the ARS of Sato both can be classified as virtual, mixed and/or 
augmented reality systems and, moreover, share several structural and functional similarities. Therefore, it would 
have been obvious to one of ordinary skill in the art, at the time of the applicant's claimed invention, to integrate the 
ARS of Sato into a vehicle, such as a military helicopter, as suggested by Hamilton et al. As suggested by Hamilton 
et al. (Hamilton et al. column 3, lines 57-61) A motivation for such an application of Sato's ARS would have been 
to allow the crew situational awareness even when confronted with low visibility or otherwise difficult flight 
conditions. Combining the teachings of Sato and Hamilton et al., in this manner, yields an vision system that 
conforms to that which is set forth in claim 1 . 

16. The following is in regard to Claim 2, As shown above, the teachings of Sato and Hamilton et al. can be 
combined to yield an integrated vision system that satisfies all aspects of claim 1 . Both Sato and Hamilton et al. 
employ HMDs in their vision systems. Hamilton et al. further include panoramic 1 view data in the HMD of their 
virtual image display system (Hamilton et al. column 10, lines 23-27), The advantage of using panoramic view data 
in a virtual, augmented or mixed reality system is that it provides a complete, continuous view of the scene or 
rendered objects from any viewing direction. Therefore, it would have been obvious to one of ordinary skill in the 
art, at the time of the applicant's claimed invention, to generate panoramic view data, as suggested by Hamilton et al. 
(and well-known in the art) and incorporate that into the HMD of the vision system obtained by combining the 
teachings of Hamilton et al. and Sato in the manner discussed above. In doing so, one would have obtained a method 
that conforms to the integrated vision system of claim 2. 

1 7. The following is in regard to Claim 3. As shown above, the teachings of Sato and Hamilton et al. can be 
combined to yield an integrated vision system that satisfies all aspects of claim 1. Both Sato (e.g. Sato column 16 
lines 35-43, column 16 lines 59-61, and column 17 lines 1-13) and Hamilton et al. (e.g. Hamilton et al. Fig. 7 and 



Note that peripheral wide-area view data will be interpreted here as being panoramic view data, as this is a term more descriptive and 
frequently used in descriptions of virtual/augmented reality systems. 
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column 4, lines 50-59), include in their systems a HMD for overlapping the visible images of the integrated view 
data (i.e. the 3D representations of the objects in the scene) and actual view. Sato additionally suggests the usage of 
a see-through HMD, as utilized in several prior art augmented reality systems (Sato column 1, lines 13-23 and 
column 8, lines 23-28). Overlaying virtual objects over real scenes has the advantage of providing a more natural 
visual environment in which the crew member can operate the vehicle. Furthermore, this obviates the computational 
burden of generating a realistic, fully-virtual environment. Taking this into account, it is clear that the vision system, 
obtained by combining the teachings of Hamilton et al. and Sato in the manner discussed above, conforms to that 
which is proposed in claim 3. 

1 8. Claim 6 is rejected under 35 U.S.C. 103(a) as being unpatentable over Strumolo et al., in view of 
Sarangapani (U.S. Patent 6,055,042), in further view of Shoucri et al. (U.S. Patent 5,999,122) and Owens et al. 
("Passive Night Vision Sensor Comparison for Unmanned Ground Vehicle Stereo Vision Navigation"). 

19. The following is in regard to Claim 6. As shown above, Strumolo et al. disclose a vision system that is in 
accordance with claim 6. The vision system of Strumolo et al. further includes a set of infra-red stereo cameras (see 
the discussion above with respect to claim 5). However, while Strumolo et al. show the use of multiple stereo 
cameras, they fail to show or suggest the usage of at least a first stereo-camera, a second stereo-camera and a third 
stereo camera, where the second stereo camera is a milli-wave camera and the third camera is an intensifer. 
Strumolo et al. also do not show or suggest that the each of the set of cameras being selectively used in accordance 
with actual views. 

20. Sarangapani disclose an apparatus for obstacle detection in the path of a mobile machine (e.g. truck 100 in 
Sarangapani Fig.l - see Sarangapani Abstract). This apparatus uses multiple sensors (Sarangapani column 3, lines 
27-30), each suited for particular environmental conditions, object properties, and distance accuracy (Sarangapani 
column 3, lines 30-47). Sensors are selected depending on their accuracy in measuring objects at certain distances 
(Sarangapani column 3, lines 5-26 and Fig. 8, step 806). The selected sensors (either near-range of far-range) are 
weighted according to the measured environmental conditions and object characteristics (Sarangapani Fig. 8 steps 
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808 or 810, column 3 lines 60-65, and column 5 lines 65-67 to column 6 lines 1-17). Thus, Sarangapani teaches each 
of the set of sensors/scanners (cameras) being selectively used in accordance with actual views. 

21. Sarangapani and Strumolo et al. are combinable because they are analogous art, that is, both are related to 
object/obstacle detection. Therefore, it would have been obvious to one of ordinary skill in the art, at the time of the 
applicant's claimed invention, to extend the functionality of Strumolo et al.'s system to include multiple sensors or 
cameras and selectively operate those cameras, in the manner of Sarangapani, according to the conditions observed 
by the sensors or cameras (i.e. the actual image). Clearly, the advantage of doing this would have been to make the 
system thus obtained adaptive to change conditions detected by any of the various cameras. 

22. Though the set of sensors suggested by Sarangapani includes, though is not limited to, radar scanners 
(Sarangapani, column 3, lines 38-44), infrared scanners and optical scanners (Sarangapani column 3, lines 27-29), 
neither Strumolo et al. nor Sarangapani suggest that the radar sensors be milli-wave radar sensors, nor do they 
suggest that the optical scanners be intensifiers. 

23. According to Shoucri et al., "[virtually any type of imaging system that can benefit by providing quality 
images under low visibility conditions could benefit by using millimeter- wave imaging". In particular, Shoucri et al. 
suggest that millimeter-wave imaging has particular applicability to collision detection and avoidance systems. See 
Shoucri et al. Discussion of the Related Art, paragraph 1. This reflects what was well-known at the time of the 
Applicant's claimed invention. Furthermore, it was well-known at the time of the Applicant's claimed invention that 
such millimeter-wave imaging systems typically utilize passive millimeter- wave cameras. 

24. These teachings of Shoucri et al. and those of Sarangapani and Strumolo et al. are combinable because they 
are analogous art. In particular, the cited teachings of Shoucri et al. and the teachings of Sarangapani and Strumolo 
et al. are all directed toward the detection of obstacles and/or collision avoidance. Therefore, it would have been 
obvious to one of ordinary skill in the art, at the time of the applicant's claimed invention, to use a millimeter- wave 
camera as one of the set of sensors in the system, obtained by the combination of Sarangapani and Strumolo et al 
Following from the discussion above, adding such a millimeter- wave camera to a vision system, involving the 
obstacle detection (e.g. the systems of Sarangapani and Strumolo et al.) would have been advantageous because such 
cameras can provide quality images under adverse visibility conditions. Thus, the combination of Shoucri et al., 
Sarangapani and Strumolo et al. yields a system, in accordance with claim 1, further comprising a multitude of 
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cameras (or sensors), say a first, second and third sensor, etc., wherein the first sensor is an infra-red stereo camera 
(Strumolo et al.), the second sensor is a millimeter- wave camera and each of the multitude of sensors, or cameras, is 
selectively used in accordance with actual views. However, neither of Shoucri et al., Sarangapani ,and Strumolo et 
al. teach that a one of the multitude of cameras or sensors, say the third camera, be an intensifier. 

25. Owens et al. discuss a night- vision stereo camera (sensor) 2 , consisting of cooled FLIRs, deployed on an 
unmanned ground vehicle (XUV). See Owens et al. Abstract and Introduction. According to Owens et al. (e.g. last 
sentence of Summary and Conclusion on page 13 1 of Owens et al.), "Only cooled COTS [commercial off the shelf] 
night vision sensors have the necessary sensitivity, exposure speed and synchronization capability to be used 
successfully as nighttime stereo vision cameras on board moving XUVs". 

26. The teachings of Owens et al. are combinable with those of Shoucri et al., Sarangapani ,and Strumolo et al. 
because they are analogous art. In particular, Shoucri et al. (those cited above), Sarangapani , Strumolo et al., and 
Owens et al. are directed to obstacle detection and/or collision avoidance. Furthermore, these teachings all have 
demonstrated applicability to ground-based vehicles. Therefore, it would have been obvious to one of ordinary skill 
in the art, at the time of the applicant's claimed invention, to use a night-vision stereo sensor, consisting of cooled 
FLIRs, as one of the set of sensors in the system, obtained by the combination of Sarangapani , Strumolo et al., and 
Shoucri et al. According to Owens et al (e.g. last sentence of Summary and Conclusion on page 13 1 of Owens et 
al.) such a modification would have been advantageous because, "[o]nly cooled COTS [commercial off the shelf] 
night vision sensors have the necessary sensitivity, exposure speed and synchronization capability to be used 
successfully as nighttime stereo vision cameras on board moving XUVs". Combining the teachings of Strumolo et 
al., Sarangapani, Shoucri et al., and Owens et al., in the manner described above, yields a vision system, in 
accordance with claim 1, further comprising at least a first stereo-camera, a second stereo-camera and a third stereo 
camera, the first stereo camera being an infrared camera (Strumolo et al), the second stereo camera being a 
millimeter-wave camera (Shoucri et al.) and the third stereo camera being an intensifier (Owens et al.), the first, the 
second and the third stereo-cameras being selectively used in accordance with actual views (Sarangapani). Such a 
system satisfies the limitations of claim 6. 



2 Night-vision sensors, be it a stereo sensor, consisting of cooled FLIRs, or otherwise, will be considered here as an intensifier, since these 
intensify, in some manner, the sensitivity of the sensor to the low-intensity light of night. 
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Any inquiry concerning this communication or earlier communications from the examiner should be 
directed to Kevin Siangchin whose telephone number is (703)305-7569. The examiner can normally be reached on 
9:00am - 5:30pm, Monday - Friday. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Amelia Au can 
be reached on (703)308-6604. The fax phone number for the organization where this application or proceeding is 
assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent Application Information 
Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR 
or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more 
information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the 
Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 
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